Award or Reward? Which comes first, NIH funding or research impact?

Common sense suggests that increasing research funding increases research productivity. We also know that with increased productivity, the odds of receiving funding increases. So, which comes first, productivity or funding? To answer this question we used causal inference (1,2) to analyze data that was recently released (3) on over 70,000 principal investigators who had received NIH funding between 1996 and 2014. We find that research productivity, as measured by citation numbers, predicts the level of NIH funding. However, the reverse is not true. There appears to be no simple monotonic relationship between the level of funding and the distribution of research productivity (Fig. 1). As measures of research productivity and funding level, we used metrics previously agreed upon by an NIH working group (3). These are the relative citation ratio (RCR), which measures the number of yearly citations of an investigator relative to the number of citations by their peers in the same research field, and the grant support index (GSI), which adds up NIH grants based on a point system. Given the importance of research funding and productivity, it is no surprise that these and other metrics are vigorously debated. We do not endorse, nor question, the merits of these metrics, but simply analyze them to determine which of two alternative causations is better supported by these data. Specifically, productivity determines funding (Hypothesis 1: RCR->GSI), or funding determines productivity (Hypothesis 2: GSI->RCR). The joint distribution of funding and productivity across investigators is shown in Fig. 1A. Causal inference applies Occam’s razor to such observational data as follows: If a given value of variable A straightforwardly determines the distribution of variable B, but there is no simple model for the distribution of variable A given B, then the hypothesis that A causes B is favored, over the hypothesis that B causes A (1,2). To explore hypothesis 1 with an established causal inference method (2), consider fig. 1B, which shows the mean of GSI conditioned on RCR. Evidently, there is a smooth monotonic increase in average NIH research funding (GSI) as research productivity (RCR) increases. The inverse is not true. As research funding increases, citations averaged across investigators do not follow a smooth monotonic increase (Fig. 1E). When we look at the spread around these mean values, we see that they also increase (Fig. 1C & F). That is, with increasing success, uncertainty also increases. If the spread we observe is due to chance, e.g. the result of a lucky or unlucky year, then we expect that the standard deviation is tightly linked to and monotonically increases with the mean (which is a characteristic of many stochastic processes; see Methods). This is precisely what we observe for funding levels (Fig. 1D) but not for research productivity (Fig. 1G). To summarize, there is a simple model for the hypothesis that citations determine funding, but no simple model to explain the behavior of citations as a function of funding (for a formal hypothesis test see Methods). A similar relationship is found when considering other measures of citations and funding provided by Lauer et al. (2017) (3) (see Methods). We conclude that research